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TITLE OF THE INVENTION 

TRACKING MENUS, SYSTEM AND METHOD 
CROSS-REFERENCE TO RELATED APPLICATION(S) 

[0001] This application is related to and claims priority to U.S. provisional application 
entitled Tracking Menu System (1252.1082P) having serial number 60/419,144, by Fitzmaurice 
et al, filed October 18, 2002, this application is also related to U.S. application entitled A Pen- 
Mouse System having serial number (1252.1085), by Fitzmaurice et al, filed concurrently 

herewith and to U.S. application entitled Pan Zoom Tool having serial number 

(1252.1086), by Fitzmaurice et al, filed concurrently herewith, all of which are incorporated by 
reference herein. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0002] The present invention is directed to a tracking menu that tracks a position of a 
transducer that can be moved by a user and that allows the user to select items locally within 
the menu and, more particularly, to a menu usable with a stylus-tablet PC that tracks the 
position of the stylus. 

2. Description of the Related Art 

[0003] A user involved in the graphics industry can be required to switch between tools while 
performing graphics operations. For example the user may need to switch between 
paintbrushes or between a pan tool and a zoom tool. In keyboard/mouse based systems the 
switching can be facilitated by accelerator techniques using a particular key, such as the space 
bar, to switch between the pan and zoom tools. In systems that have a single channel input 
device, such as pen-based computers like a stylus/pen for tablet PCs, the user does not 
typically have access to accelerator techniques that are efficient or easy to use. As a result, the 
user typically moves from the location of the graphics operation on a tablet PC (personal 
computer) to a menu or tool palette location, to change tools or select a new tool. These tool 
palette round trips are time consuming. This problem also arises in related technologies such 
as personal digital assistants (PDAs), touch based interfaces, digitizer tablets, wall displays and 
3D environments that use 6D tracking devices etc. 

[0004] What is needed is a system that will avoid tool palette round trips for such limited input 
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device systems. 

SUMMARY OF THE INVENTION 

[0005] It is an aspect of the present invention to provide an interface that tracks a position of 
a cursor or transducer tracking symbol. 

[0006] It is another aspect of the present invention to provide a movement tracking interface 
that allows the user to select functions within the interface without large movements of the 
transducer. 

[0007] It is a further object of the present invention to optionally allow a user complete 
visibility of a scene when performing a selected function but have the tracking interface 
available at the cursor when needed. 

[0008] The above aspects can be attained by a system that includes a tracking menu that 
tracks the movement of a positions transducer, such as a stylus or a mouse, as the transducer 
is moved about. The menu includes a tracking symbol, such as an arrow or cursor, positioned 
corresponding to inputs from transducer as moved by a user. A mobile tracking region is also 
included. This tracking region has a boundary enclosing the tracking symbol where the tracking 
symbol is movable within the boundary. The tracking region moves in correspondence to the 
tracking symbol when the tracking symbol encounters or bumps against the boundary while 
moving. The tracking region also has controls or buttons that are activatable when the tracking 
symbol is over or corresponds to the controls. When a stylus is used as the transducer, the 
menu tracks the stylus as it moves above a display and the controls are activated when the 
stylus touches the display at a control. When the stylus moves out of tracking range, the menu 
stops tracking. When the stylus returns to tracking range the menu jumps to the new position of 
the stylus on the display. The tracking menu can take many shapes and the tracking region 
need not coincide with the visible boundary of the menu. Stylus functions can performed, such 
as painting, when the stylus touches the display in a region not corresponding to a control and 
the menu becomes invisible while tracking during function execution. Stylus functions can also 
be performed, such as pan/zoom, when the stylus moves over the display. 

[0009] These together with other aspects and advantages which will be subsequently 
apparent, reside in the details of construction and operation as more fully hereinafter described 
and claimed, reference being had to the accompanying drawings forming a part hereof, wherein 
like numerals refer to like parts throughout. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0010] Figure 1 depicts a tracking menu according to the present invention. 

[001 1] Figures 2a-2c illustrate behavior of the tracking menu. 

[0012] Figure 3 illustrates components of the tracking menu. 

[0013] Figure 4 depicts events associated with a stylus and tablet. 

[0014] Figure 5 is a state diagram of the tracking menu. 

[001 5] Figures 6a-6f depict features of a tracking menu. 

[0016] Figure 7 illustrates tracking operations. 

[0017] Figures 8a and 8b depict a tracking menu with a pin capability. 

[0018] Figure 9 depicts tracking operations including pinning. 

[0019] Figure 10 depicts tracking operations with a mouse having a tracking symbol. 

[0020] Figures 1 1 a-1 1 c depict behavior of a tracking symbol with a persistent object. 

[0021] Figure 12 depicts operations associated with a persistent object. 

[0022] Figures 13a-13c depict a tracking menu deforming during movement. 

[0023] Figures 14a and 14b show different internal region arrangements. 

[0024] Figure 15 depicts non-coincidence between a visible menu boundary and a tracking 
boundary. 

[0025] Figures 16a and 16b depict additional non-coincidence between visible menu 
boundary and tracking boundary. 

[0026] Figure 17 shows an interior tracking boundary 

[0027] Figure 1 8 shows interior tracking walls or partitions. 

[0028] Figure 19 shows a desktop PC as hardware of the present invention. 

[0029] Figure 20 depicts a tablet PC as hardware of the present invention. 

[0030] Figure 21 shows components of the hardware of the present invention. 

[0031] Figure 22 shows a mouse shaped tracking menu. 

[0032] Figure 23 shows composition of two tracking menus. 
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[0033] Figures 24, 25, 26, 27, 28, 29 and 30 show color palette, linear, numeric pad, graffiti, 
keyboard, marking tracking menus, and 3D view controls, respectively. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0034] The present invention is directed to an interface that tracks the movement of a stylus, 
finger, mouse, etc. and allows the user to switch tools without the need to move to a typical 
display menu, such as a menu bar at a top of a display. This interface is called a tracking menu. 
A tracking menu 10, as depicted in figure 1, is a graphical user interface (GUI) that includes 
mobile tracking region 1 1 typically having a cluster of controls or graphical buttons 12 and 13, 
and as with traditional menus, a tracking symbol or cursor 14 can be moved within or over the 
menu to select and interact with items, such as the buttons. However, unlike traditional menus, 
when the cursor 14 hits or crosses an edge or boundary 15 of the tracking region 1 1 of the 
menu 10, the menu 10 moves to continue tracking the cursor 14. Thus, the menu 10 always 
stays under the cursor 14 and close at hand. The tracking menu is invoked and dismissed in 
the same manner as a traditional modal tool by clicking on a tool palette or menu item. 

[0035] Figure 2a shows an arrow shaped tracking symbol type cursor 14 moving within the 
menu 10 in the mobile tracking or exterior region 1 1 from the position depicted in figure 1 to a 
new position along a path depicted by the cursor trail 16. Figure 2b shows the movement of the 
cursor 14 continuing to move along a path depicted by trail 18 over button 13 where the button 
13 is conventionally highlighted. Figure 2c shows the movement of the cursor 14 along a path 
depicted by cursor trail 20 until the cursor 14 encounters a right boundary or edge 22 of the 
menu 10 which results in the menu 10 moving to the right as the cursor 14 keeps moving to the 
right, as depicted by the menu object trails 24. As a result, the menu 10 also tracks the position 
of the transducer and is itself a tracking symbol or a mobile control. 

[0036] The behavior of the tracking menu 10, as discussed above, can be understood by a 
simple analogy. Consider moving a jar lid with the tip of a pencil. This can be done in two 
ways. The first simple way is that the pencil can be pressed down into the lid with sufficient 
force that the lid dragged as the pencil is moved. In this motion the point of contact with 
between the lid and the pencil does not change. The second way, which is used in the tracking 
menu of the present invention, is by moving the pencil in the air above the lid with the pencil 
inside the edge of the lid and moving the lid when the pencil contacts the sides of the lid. This 
results in the ability to move the lid without pressing down. Note that the pencil can be moved 
within the lid as well and the lid will remain stationary if the sides are not contacted. 
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[0037] From the lid analogy point of view, with respect to a graphic desktop 40 (see figure 3) 
the tracking menu 42 of the present invention has several components as depicted in figure 3. 
The cursor 44 is allowed to move or float inside the menu 42 until it encounters a tracking menu 
edge 46 at which point the menu 42 tracks the cursor 44 until the cursor moves away from the 
edge 46. 

[0038] The present invention preferably activates and controls tracking menus using the 
conventional multiple input states sensed by pen computers (or using regular mouse events in 
standard keyboard and mouse configurations). Figure 4 illustrates the pen input states sensed 
by a conventional tablet PC 60 and used by the present invention. When the stylus or pen 62a 
is more than approximately 1 .5 cm above the tablet surface 64, it is out-of-range 66 and the 
Tablet PC system does not track the location of the pen 62. When the pen 62b is moved closer, 
the tablet tracking system 60 begins tracking 68 the tip of the pen 62b and the cursor (not 
shown) on the display of the tablet 60 follows the tip of the pen. Finally, touching 70 occurs 
when the pen 62c contacts the tablet surface 64 and the tablet system 60 detects the touching. 

[0039] The operation of the tracking menu of the present invention can be understood using 
a state diagram as depicted in figure 5. In state 0 the pen is out-of-range (66) of the tracking 
system of the tablet PC and the tracking menu is visible (preferably semi-transparent) and is 
positioned in the last place it resided before the pen moved out-of-range or in an initial position if 
the tracking menu has just been invoked. The state 0 is entered from states 1 and 1E when the 
pen is lifted from the tablet and moved out-of-range. Preferably the menu resides in this last 
position until the pen is brought back into range and the tablet PC begins tracking (68) the pen. 
However, the menu could continue moving even when the stylus is out of tracking range (e.g., a 
"momentum" algorithm could be used where the menu moves as if it were floating in a viscous 
fluid and when force no longer is being applied, eventually slows down). 

[0040] The pen can move into tracking range and enter state 1 in two different ways. In the 
first into-range transition, the pen comes down at a position where the tip of the pen is outside of 
the tracking menu edge (46). When tracking starts in this condition, the menu is moved or 
repositioned 90 so that the cursor is inside the menu edge. This involves conventionally 
redrawing the semi-transparent menu at a position corresponding to cursor. This repositioning 
or redrawing can place the menu such that the menu is moved the minimum distance to place 
the cursor just within the edge of the menu or a least Euclidean distance from the prior position. 
Or the repositioning can place the menu at an arbitrary position under the cursor, such as 
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positioning the menu with the pen tip in the center of the menu. The second into-range 
transition occurs when the pen tip comes into range while it is within the boundary (46) of the 
tracking menu. 

[0041] In state 1 the cursor moves freely about within the menu and the menu stays 
stationary. During this movement of the cursor within the menu, the system performs 
conventional operations, such as highlighting buttons or controls over which the cursor passes 
by comparing the position of the cursor to positions of the buttons. However, if the pen is lifted 
out-of-range the state moves to state 0 (pen tracking becomes inactive), if the pen encounters 
an edge as it moves, state 1E is entered and if the pen touches the tablet state 2 is entered. 

[0042] To enter state 1 E the position of the cursor is conventionally compared to the 
position of the edges of the tracking menu. When a match occurs, the cursor has hit the edge 
and the state 1 E is entered. In state 1 E, as long as the cursor is at or in contact with an edge as 
the cursor moves, the tracking menu (semi-transparent) is moved along with the cursor. That is, 
as the cursor is moved, the menu is conventionally redrawn with the cursor at the edge of the 
tracking menu. In state 1E, if the pen is lifted out-of-range the state moves to state 0, if the pen 
moves away from an edge to reenter the interior of the menu the state moves to state 1 and if 
the pen touches the tablet, state 2 is entered. 

[0043] As discussed above, state 2 is entered when the pen touches (70) the tablet surface 
while in state 1 or state 1E. In this state the pen can be active or activated such that it will 
cause some function to be performed. It is also possible there could be no functionality 
assigned to this region of the tracking menu. Here the tracking menu may not even change its 
appearance. In state 2 the active pen can be selecting a button, in which case the function of 
the button is performed, such as selection of a new tool. Or the active pen can be moving while 
under the control of a previously selected function, such as painting with a previously selected 
paintbrush or zooming based on a previous selection of a zoom tool/function. In state 2, the 
tracking menu can be made fully transparent or "disappear". (When in state 2, it is optional 
whether the tracking menu "disappears". Instead the visuals of the menu can change, for 
example, button highlighting changes to indicate it is pushed in or a check box toggles). In this 
state, the system can continue to reposition the fully transparent menu under the cursor or 
preferably the menu can be allowed to remain in it's last position as in state 0 (note the user 
would not perceive a difference between these two alternatives). When the pen is lifted from 
the tablet surface and contact with the tablet ends, the tracking mode 68 is again entered and 
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the menu is repositioned 92 depending on the last state. If the pen is lifted when the prior state 
is state 1 E, the pen is repositioned 92 at the last edge point of state 1 E. If the pen is lifted when 
the prior state is state 1 , the pen is repositioned 92 at the last interior point of state 1 . The 
above discussed repositioning strategies are best practice or preferred. Other practices may be 
employed depending on the application of tracking menus. For example, repositioning a 
tracking menu within a spreadsheet application may reposition into the nearest spreadsheet cell 
or row/column. Other strategies may have the tracking menu always appear at the top-left of 
the screen. 

[0044] When using a mouse, the same set of state transitions apply except that out-of-range 
(state 0) does not occur. Rather than state 0, the system includes the functionality of directly 
jumping to a new screen position. However, this result can also be achieved by moving the 
tracking menu in state 1E. Thus, tracking menus work both with a pen and a mouse. 

[0045] Figures 6a-6f visually depict the operations discussed above. Figure 6a depicts a 
tracking menu in an initial state where the user selects a pencil tool by moving the cursor 82 
over the pencil button 84 and performing a pen down. In figure 6b the pencil tool is assigned to 
the exterior region of tracking menu 80 and the cursor 82 changes to a pencil icon. In figure 6c 
the user places the pen/pencil down, touching the table PC surface while in the exterior region 
80 and the tracking menu becomes invisible. At this point, a pencil drag operation is performed 
that results in making a mark 86. In figure 6d the tracking menu 80 reappears on a pen-up 
event and the menu 80 is repositioned under the cursor 82. In figure 6e the user selects a 
second tool, the flood fill tool. In this example, the exterior region of the tracking menu is 
divided or split into two regions 90 and 92 by providing conventional user enabling assignment 
controls for each side. When cursor moves to the left side of the tracking menu as shown in 
figure 6f, the pencil tool is enabled and cursor changes to the pencil icon. Conversely, when the 
cursor moves to the right side, flood fill tool and corresponding icon are activated. 

[0046] An alternate approach to describing the operations of the present invention and one 
that also describes additional features of the invention is set forth below with respect to figures 
7, 9 and 10. 

[0047] Tracking operations are described with respect to figure 7 and these operations are 
performed while the tablet PC or conventional PC is tracking 1 10 the position of an input 
transducer, such as a stylus or mouse respectively, during which the tracking menu is displayed 
112. As the transducer is moved, the system determines 1 14 whether the display location 
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associated with input transducer is within the boundary of the tracking menu. If so, the system 
executes 116 any location specific behavior that is active. If not, the system determines 1 18 
whether the display location is near the menu border. If so, the position of the menu is adjusted 
120. If the location is not near the border, the new menu placement position is determined 122 
based on a predetermined placement strategy. The strategies include the centering and 
minimum displacement strategies previously discussed and the positioning of the menu so that 
the cursor is over a specific button or element of the menu. Once the new position is 
determined, the menu is moved 124 to the new position where the movement could be a jump, 
animation or some other form of transition. Additional tracking operations with respect to state 2 
are described below with respect to figure 10. 

[0048] Depending on the user's workflow, it may be desirable to separate the cursor from 
the tracking menu. For example, a user may desire the ability to rapidly switch between the 
pan-and-zoom tool and a drawing tool. To accommodate this type of feature, the tracking menu 
130 as depicted in figures 8a and 8b includes a pushpin. When the user selects the pushpin 
button 132, the tracking menu 130 is temporarily deactivated, remaining posted and stationary; 
and it grays out to indicate the inactive state (see figure 8b). The cursor 134 can now leave or 
pass over the tracking boundary edge 136. The next time the cursor 134 travels into the 
tracking menu 130 the pushpin is automatically released and the tracking menu 130 behaves 
normally (i.e., moving when the cursor hits the tracking border). A lock, which can also be 
implemented with a corresponding icon, explicitly pins the tracking menu and does not release 
the menu until the lock is explicitly selected again, even when the tracking symbol 134 crosses 
into the menu 130. 

[0049] Figure 9 depicts the tracking operations of figure 7 and also includes operations that 
allow a menu to be pinned (and locked) in place. During the tracking and display of the menu 
the system determines 144 whether the menu has been pinned. If so, the system determines 
146 whether the transducer display location is outside of the boundary of the menu. If so, no 
change in the pinning mode is made. If the location is outside the menu, tracking of the display 
location of the transducer (cursor) while the menu is pinned continues. If the location is inside 
the menu, the system checks 148 to see if the menu is locked down. If so, tracking while the 
menu is pinned continues. If the menu is not locked down, the menu is unpinned 150 and then 
tracking resumes where the menu follows the position of the cursor. 

[0050] Figure 10 illustrates tracking operations that occur when a mouse is being used and 
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continuous or one-shot events are activated by a mouse down or a mouse button activation 
event is detected. Tracking of the display position of the mouse cursor, such as an arrow, 
occurs 160 and continues even when a mouse down (button activation) occurs. When a mouse 
down event occurs, the system determines 162 whether the location is on the menu. If so, the 
system determines 164 the subcomponent of the menu associated with the cursor location and 
changes 166 the appearance of the subcomponent, such as by changing in the appearance of a 
button to indicate that it has been activated. If the subcomponent activated by the mouse down 
event is a one-shot type of control 166, the system performs the operation responsive to the 
down event or responsive to an up event where the mouse button is released. The appearance 
of the control can be changed to show that the one-shot operation is being performed. Once 
the operation is performed the system returns to tracking and menu display (not shown). If the 
selected control is a continuous operation type control, the continuous operation is activated 
and the tracking symbol can be changed 170 to reflect the change in mode. During continuous 
operation activation, the user menus can be displayed or not responsive to a preset menu 
display option. The system tests 174 for this option and redraws 176 the display without the 
menu if the option is active. The system then performs 178 the continuous operation of the 
activated control as the cursor is moved. On a mouse button up event 180, the menu is 
repositioned, the tracking symbol or cursor is restored if it has been changed and the menu can 
be animated to transition to its new location. 

[0051] When the tracking menu is moved about in the display it is preferably drawn 
(semitransparent) on top of most other objects in the display, such as a model being drawn or 
manipulated. However, some objects in the display space take precedence over the tracking 
menu. For example, an operating system menu bar at a top, side or bottom of a display may 
need to be persistently displayed even when the tracking menu moves into the display space of 
the menu bar. In such a case it is preferred that the tracking menu change to a display state in 
which the presence of the menu is graphically shown but the predominance of the persistent 
object is apparent. An example of such a preference is shown in figures 11a-11c. Figure 11a 
depicts a pan-zoom type tracking menu 210 being moved (dragged) by cursor 212 toward a 
persistent menu 214 in a corner of a display 216. The tracking menu 210 is displayed or drawn 
showing the complete graphics of the menu. When the cursor crosses the boundary of the 
persistent menu 214, as depicted in figure 11b, the tracking menu is converted into an x-ray, 
shadow or outline type menu 218. This menu display mode displays an outline 218 of the menu 
over the persistent object (214) and within the display space not occupied by the persistent 
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object, if necessary, to provide the user with a visual impression of the tracking menu existence. 
This also can provide the impression of the tracking menu sliding under the persistent object. 
This outline menu 218 can also depict graphic symbols of the tracking menu in outline form or in 
non-outline form to assist in this visual impression. For example, figure 11b shows a pan control 
symbol 220 in non-outline form. While the tracking menu is in the outline mode any events 
initiated by the transducer, such as a touch of the display by a pen or a mouse down event are 
interpreted as events for the persistent object 214 and provided to a control process of the 
persistent object. As a result, the tracking menu process enters a non-tracking condition similar 
to the out-of-range condition 66 previously discussed. When the cursor 212 moves back 
outside the persistent object (214), as depicted in figure 11c, the display mode for the tracking 
menu changes back to the non-outline or complete graphic mode menu 210 where the graphic 
details of the tracking menu are shown. This can appear as if the tracking menu is sliding out 
from under the persistent object. The menu acts like the transducer has come within range and 
tracking begins (68) with the menu being positioned responsive to the placement strategy in 
effect. In this exit operation, the persistent menu is drawn last so that portions of the tracking 
menu "under" the persistent object are not shown or the overlap portion of the tracking menu 
210 is clipped. 

[0052] Note that when the transducer is over the persistent object and the tracking menu is 
underneath the persistent object, events can also be first processed by the persistent object 
and, optionally, passed through to the tracking menu for it to process. In addition, the persistent 
object can be used to re-assign functionality to the underlying tracking menu since the objects 
are stacked. For example, in Figure 1 1 b, if the user wanted to assign the outer region of the 
tracking menu, which, for example, is currently the pan tool, to a different tool, say, the "reverse" 
tool, the user could move the tracking menu under the persistent object and the act of selecting 
the "reverse" tool would assign the functionality to the tracking menu region below the current 
cursor location. 

[0053] Figure 12 depicts the operations of the outline menu process discussed above. 
While menu tracking is being performed 240, the system determines 242 whether the cursor 
has moved over a persistent object. If so, the tracking menu is converted 244 to the outline 
mode, followed by deactivating menu tracking 246 and activating the persistent object, such as 
a menu 214. Then all transducer events are passed 250 to the persistent object process until 
the cursor travels back outside the persistent object. When the cursor is detected as being not 
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over the persistent object, the system determines 252 whether the previous position of the 
cursor was over the persistent object. If this condition exists, the cursor has moved out of the 
persistent object and the position of the display location is determined 254 and a placement 
strategy is invoked. The tracking menu is converted 256 into the complete graphic display. The 
tools of the tracking menu are activated 258 and events are no longer sent to the persistent 
object process resulting in the persistent object being deactivated. The tracking menu is then 
clipped 260 as needed and tracking 240 resumes. 

[0054] As noted above, when the tracking menu encounters a persistent object the menu 
can be made to appear to slide under the persistent object. It is also possible to deform the 
tracking menu as it encounters a persistent object or a display edge or corner. Figures 13a-13c 
depict a tracking menu having a graphic in the form of a mouse 250 with a circular tracking 
boundary 252 deforming as it is dragged to the right (figure 13b) and then being deformed to fit 
a corner 254. This deformation will allow the tracking menu to be used throughout an available 
space of a display. 

[0055] While moving the tracking menu in the tracking input state, a variety of dragging 
algorithms can be employed. The above discussion describes the simple physical approach 
that moves the tracking menu at the point of cursor contact with the tracking menu edge and 
keeps the cursor stuck at the edge until the user "backs up" a bit. Alternatively, the tracking 
menu process could use a different dragging algorithm such that the cursor gets attached to the 
tracking menu edge but can go beyond the edge and drags the tracking menu through a 
metaphorical string or elastic. Simulating gravity and weight for the tracking menu and 
imparting forces through cursor activity is possible and may add a fun factor to the technique. 

[0056] There are a variety of ways to divide the exterior region of the tracking menu as 
depicted in figures 14a and 14b. The menus 270 and 280 of figures 14a and 14b show how 
regions can be laid out so that some functions are easy to invoke by being placed against the 
edge of the tracking menu. This characteristic allows selection by direction of movement rather 
than only by position, such as in marking menus. 

[0057] The visual boundary of the graphical representation of the tracking menu does not 
have to map directly to the tracking boundary as depicted in figure 15. The tracking boundary 
can have a different shape and it can be larger or smaller than the visual boundary. Figure 15 
shows a circular graphical visible tracking menu 290 with an elliptical border or edge 292. 
Figure 16a shows a tracking menu in which the tracking boundary 294 is inside the visual 
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boundary. Figure 16b depicts a tracking boundary 298 partially inside and partially outside the 
visual boundary 300. Moreover, tracking boundaries can be non-contiguous. For example, 
there could be a hole 302 in the tracking menu 304 as depicted in figure 17 (see also figure 26) 
or the menu 306 can include interior tracking menu boundaries, partitions or walls 308 and 310 
that jut out from the exterior boundary 306 as depicted in figure 18. When the cursor 312 
encounters an interior wall 308, the menu 306 behaves as if it has encountered an edge and the 
menu is moved when the cursor "pushes" against the wall. Interior walls may be useful to bias 
the space and allow the cursor to remain in a sub-region more easily. A wall can be used to 
rotate the menu like a track ball type operation by pushing against the wall and moving in a 
circular direction. This rotation can also be produced without a wall if the edge is made "sticky". 

[0058] The hardware of the tracking menu system can be within desktop PC 330 (see figure 
19) or within a handheld device, such as a tablet PC 340 (see figure 20) or a PDA, and includes 
an input transducer 360 the position of which is tracked by a computer 362 that processes the 
transducer positions and creates the tracking menu display that is presented on a display 364 
as shown in figure 21. 

[0059] The tracking menu of the present invention can take a number of different shapes 
with correspondingly different functions and capabilities. As discussed above, the tracking 
menu can have a circular shape and perform as a pan-zoom tool. A pan-zoom tool tracking 
menu is described in more detail in the related application noted above. The tracking menu can 
also take the shape of and act as a mouse or pen-mouse 380 as discussed above and depicted 
in figure 22. A pen-mouse tracking menu is described in more detail in the related application 
noted above. Two or more tracking menus 400 and 402 can be composed, attached or 
combined as depicted in figure 23. Tracking menus can allow non-uniform movement of the 
tracking symbol within the menu or can snap to interior grid lines during movement. A tracking 
menu can be a color pallet 420 including a cross-hair tracking symbol, as depicted in figure 24. 
A tracking menu can take the form of a linear menu 422 as depicted in figure 25. A tracking 
menu can be various versions of a numeric pad such as a calculator 424 where input events 
over number buttons send numbers to the numbers field and input event events over 
operations, such as add ("+") cause a corresponding operation (see figure 26). A tracking menu 
can be a graffiti input GUI 426 as in figure 27 or a display keyboard with an input region 428 as 
depicted in figure 28. The tracking menu can also include a marking menu 430 activatable by 
an embedded button as depicted in figure 29. Figure 30 shows a tracking menu 432 for 
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controlling the position and orientation of a viewpoint in a 3D scene. Typically, this is called a 
3D virtual camera and involves several separate tools for panning, zooming, and tumbling 
(orbiting the camera about the center of the 3D scene). Furthermore, there are other types of 
camera movements that can be used such as roll, yaw, and pitch. These controls form a cluster 
of functionality that can be made available via a tracking menu. Figure 30 shows a tracking 
menu where the most frequently used camera control (tumbling) is given priority in the design 
by being placed in the large outer region. Additional, less frequently used commands are 
placed appropriately in smaller regions thus reducing the chance of accidental engagement. 
Note that this design explores the usage of three permeable zones (reset view, undo and redo). 
Here the user must dwell over the region border with the cursor and after some time 
(approximately half a second) may enter and activate the zone. This provides a way of offering 
functionality within the tracking menu but at a reduced level of accessibility. 

[0060] The functionality assigned to a particular region can change based on where the 
tracking menu is on the screen and specifically when it is over an object (i.e., context sensitive). 
For example, consider a pan-and-scale tracking menu. It looks similar to the pan and zoom 
menu (or widget) except the center zone performs an object scaling operation. If the center 
zone is near the object's corners or center of the object, it does a proportional scale of the entire 
object (e.g., both x and y). If the center zone is near an edge, it scales in that direction only 
(e.g., only x or only y). The system detects the position of the menu, compares the menu 
position to object position and changes the functionality of controls accordingly. 

[0061] The present invention has been described with respect to the use of the invention 
with tablet type personal computers. It is also possible to use the present invention with other 
types of display systems, such as large format displays (plasma screens and projectors; 
front/rear projection; horizontal and vertical surfaces) - for example, white-board applications. 

[0062] The system also includes permanent or removable storage, such as magnetic and 
optical discs, RAM, ROM, etc. on which the process and data structures of the present invention 
can be stored and distributed. The processes can also be distributed via, for example, 
downloading over a network such as the Internet. 

[0063] The many features and advantages of the invention are apparent from the detailed 
specification and, thus, it is intended by the appended claims to cover all such features and 
advantages of the invention that fall within the true spirit and scope of the invention. Further, 
since numerous modifications and changes will readily occur to those skilled in the art, it is not 
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desired to limit the invention to the exact construction and operation illustrated and described, 
and accordingly all suitable modifications and equivalents may be resorted to, falling within the 
scope of the invention. 
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